3d model updates using crowdsourced video

ABSTRACT

An exemplary method includes prompting a user to capture video data at a location. The location is associated with navigation directions for the user. Information representing visual orientation and positioning information associated with the captured video data is received by one or more computing devices, and a stored data model representing a 3D geometry depicting objects associated with the location is accessed. Between corresponding images from the captured video data and projections of the 3D geometry, one or more candidate change regions are detected. Each candidate change region indicates an area of visual difference between the captured video data and projections. When it is detected that a count of the one or more candidate change regions is below a threshold, the stored model data is updated with at least part of the captured video data based on the visual orientation and positioning information associated with the captured video data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/339,966, filed on Jul. 24, 2014, which is a continuation ofU.S. patent application Ser. No. 14/055,132, filed Oct. 16, 2013, andissued as U.S. Pat. No. 8,818,081 on Aug. 26, 2014, the disclosure ofwhich is incorporated herein by reference.

BACKGROUND

Real-world 3D models may depict objects, such as building and monuments,located at some remote or interesting geographic location. Typically,current techniques to create real-world 3D models are based onlarge-scale collection efforts (e.g., aerial and other imagery collectedprofessionally). However, this can be a costly and time-consumingprocess. Moreover, these 3D models can become stale and inaccurateovertime due to changes at the location after the collections arecomplete.

BRIEF SUMMARY

Aspects of the disclosure may be advantageous for providing a scalablemethod of updating a 3D model of a location by using crowd-sourcedimaging data. For example, by receiving video segments from a largenumber of users at a location, an accurate and up-to-date 3D model ofthat location can be maintained.

One aspect of the present technology provides a method that includesusing one or more computing devices to prompt a user to capture videodata at a location. The location may be associated with navigationdirections for the user. The computing devices may receive the capturedvideo data including information representing visual orientation andpositioning information associated with the captured video data, and astored data model representing a 3D geometry depicting objectsassociated with the location may be accessed. Using the computingdevices, one or more candidate change regions may be detected betweencorresponding images from the captured video data and projections of the3D geometry. Each candidate change region indicates an area of visualdifference between the captured video data and the projections. When thecomputing devices detect that a count of the one or more candidatechange regions is below a threshold, the stored model data may beupdated with at least part of the captured video data based on thevisual orientation and positioning information associated with thecaptured video data.

In one example, the positioning information associated with the capturedvideo data overlaps with positioning information for the 3D geometry. Inthis example, a visual correlation may be determined between one or moreobjects depicted in the captured video data and one or more objectsdepicted in the projections.

In another example, detecting includes comparing features in a certainarea of a particular image from the captured video data with features ina corresponding image from the projections to obtain a difference value.

In yet another example, updating the stored model data includesintegrating images from the captured video data into the 3D geometry. Analignment of an integrated image may be determined based on acorresponding image from the projections. The alignment of theintegrated images may be adjusted using the visual orientationinformation associated with the corresponding image from theprojections.

Another aspect of the present technology provides a non-transitorycomputer readable medium including instructions that, when executed byone or more processors, cause the one or more processors to perform amethod that includes prompting a user to capture video data at alocation. The location may be associated with navigation directions forthe user. The processors may receive the video data includinginformation representing visual orientation and positioning informationassociated with the captured video data, and a stored data modelrepresenting a 3D geometry depicting objects associated with thelocation may be accessed. Using the one or more processors, one or morecandidate change regions may be detected between corresponding imagesfrom the captured video data and projections of the 3D geometry. Eachcandidate change region indicates an area of visual difference betweenthe captured video data and the projections. When the processors detectthat a count of the one or more candidate change regions is below athreshold, the stored model data may be updated with at least part ofthe captured video data based on the visual orientation and positioninginformation associated with the captured video data.

Yet another aspect of the present technology provides a system includinga memory and one or more processors coupled to the memory. The one ormore processors are configured to prompt a user to capture video data ata location. The location may be associated with navigation directionsfor the user. Information representing visual orientation andpositioning information associated with the captured video data may bereceived, and a stored data model representing a 3D geometry depictingobjects associated with the location may be accessed. Thereafter, one ormore candidate change regions may be detected between correspondingimages from the captured video data and projections of the 3D geometry.Each candidate change region indicates an area of visual differencebetween the captured video data and the projections. When the one ormore processors detect that a count of the one or more candidate changeregions is below a threshold, the stored model data may be updated withat least part of the captured video data based on the visual orientationand positioning information associated with the captured video data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an image capture process in accordance withaspects of the disclosure.

FIGS. 2A-B illustrates orientation parameters for a device in accordancewith aspects of the disclosure.

FIG. 3 is an illustration of an image processing system in accordancewith aspects of the disclosure.

FIG. 4 is an illustration of aspects of the image processing system ofFIG. 3 in accordance with aspects of the disclosure.

FIG. 5 is an illustration of an image comparison technique in accordancewith aspects of the disclosure.

FIG. 6 is an illustration of an image analysis technique in accordancewith aspects of the disclosure.

FIG. 7 is an illustration of an updating technique in accordance withaspects of the disclosure.

FIG. 8 is a pictorial diagram of a system in accordance with aspects ofthe disclosure.

FIG. 9 is a block diagram of a system in accordance with aspects of thedisclosure.

FIG. 10 is a flow diagram of a method in accordance with aspects of thedisclosure.

DETAILED DESCRIPTION

Aspects, features and advantages of the disclosure will be appreciatedwhen considered with reference to the following description ofembodiments and accompanying figures. The same reference numbers indifferent drawings may identify the same or similar elements.Furthermore, the following description is not limiting; the scope of thepresent technology is defined by the appended claims and equivalents.While certain processes in accordance with example embodiments are shownin the figures as occurring in a linear fashion, this is not arequirement unless expressly stated herein. Different processes may beperformed in a different order or concurrently. Steps may also be addedor omitted unless otherwise stated.

The present disclosure relates generally to a scalable method ofupdating a 3D model of a location by using crowd-sourced images, such asimages from a video segment. These images can provide information tocorrect, update and add to a 3D model. In some aspects, the techniquesdescribed herein may be employed as an application for use by a mobiledevice, such as a mobile phone. While aspects of the disclosure arediscussed below in connection with certain types of mobile devices, thetechniques described herein can be used in applications for other typesof mobile devices, for example, a personal computer, video camera or awearable head mounted computer that includes capabilities for capturingvideo/digital images.

In some embodiments, the user may have to install an application and/orselect a service in order to obtain the benefits of the techniquesdescribed herein. In such embodiments, if the user has not done so, theuser may not be prompted and may not have the opportunity to help updatethe model.

In one such embodiment, for example, users can receive directions to aspecific location. In this example, when the users arrive at thatlocation, they are may be asked to take a short video segment, which theapplication then uploads to a central server. This piece of videosegment, depicting objects that the users saw at the location, may beused to update the 3D model of that location. In some instances, theusers are prompted to generally move the mobile device around thelocation or to move the device in a particular way so as to capturecertain parts of the location.

By receiving video segments from a large number of users at a givenlocation, an accurate and up-to-date 3D model of the location can bemaintained. In this regard, a visual correlation can be determinedbetween one or more objects depicted in the video segments andprojections (e.g., reconstructed target images) of the 3D model. Theseprojections of the 3D model depict objects that are associated with thelocation. Thereupon, one or more candidate change regions are detectedbetween corresponding images from the video segment and projections ofthe 3D model. Each candidate change region may indicate an area ofvisual difference between the corresponding images.

If a count of the one or more candidate change regions is below acertain threshold, the 3D model is then updated with at least part ofthe video segment. For example, images from the video segment areintegrated into the 3D model based on the visual orientation andpositioning information associated with the video segment. If the countof the one or more candidate change regions meets or exceeds the certainthreshold, the video segment may not be used to update the 3D model ormay be otherwise discarded. An advantage of updating the 3D model inthis manner is to ensure that changes applied to the 3D model based onnew video segments are done gradually.

As noted above, in some embodiments, in order to obtain the benefits ofthe techniques described herein, users may be required to select asetting and/or install an application. In addition, certain data may betreated in one or more ways before it is stored or used, so thatpersonally identifiable information is removed. For example, a user'sidentity may be treated so that no personally identifiable informationcan be determined for the user, or a user's current and historicallocation may be generalized where location information is obtained (suchas to a city, ZIP code or state level), so that a particular location ofa user cannot be determined.

FIG. 1 is an illustration of an image capturing process 100 that may beemployed. In many situations, users are able to use a mobile device toget directions to the location. For example, a user may input into anapplication installed on the mobile device, a request for directions toa location, such as to famous restaurant. In return, the application maydisplay turn-by-turn directions to that location on a display screen ofthe device.

Using the same or another device, the user upon arrival at the locationmay take video segments of their surroundings. For example, as shown inFIG. 1, one or more of users 105 may use a device, such as mobile device101, to capture images of objects (e.g., buildings 102, 104 and 106) attheir location. The video segments taken by the users 105 may captureimages of people, places or things of interest at the location in anangle generally perpendicular to the ground, or where the mobile device101 is positioned at or near ground level.

In some situations, the users 105 may be prompted to capture images oftheir surroundings once they arrive at the location. For example, anapplication installed on mobile device 101 may prompt one or more of theusers 105 to take a short video segment in a general or specificdirection at the location. The images may capture objects at thelocation that may be of interest to the users 105 or other users of theapplication. The arrival of a user at the location may be detected basedon, for example, a global positioning service (e.g., GPS). In thisregard, the mobile device 101 may be equipped with GPS components thatmay be used to determine when one or more of the users 105 have arrivedat the location.

With a number of users capturing images of their surroundings at alocation, these images can be used to update and maintain a 3D modelassociated with that location. To capture the images, the user's mobiledevice may be equipped with a camera or other types of image capturingcomponents incorporated into the device. For example, mobile device 101may include a camera capable of capturing digital images of objects,such as a digital still camera, digital video camera and/or imagesensor. These images may be stored in conventional formats, such as JPEGor MPEG, or other types of formats. The images may be stored locally ina memory of the mobile device 101, such as in RAM or on a flash card.Alternatively, the images may be captured and stored remotely.

As noted above, users of mobile devices may capture images of people,places or things of interest at different locations. Typically, theimages can contain as many objects associable with the location (such asstreet lights, signs and advertisements, mountains, trees, sculptures,bodies of water, storefronts, etc.) in as much detail as may be capturedby an imaging capturing device. In addition to being associated with alocation, the images may also be associated with information indicatingthe orientation of the images. For example, the orientation of theimages may simply correspond to a camera angle, such as an angle that is30° East of true North and rises 2° from ground level. If the images are360° panoramas centered at a location, the orientation may indicate theportion of the images that correspond with looking due North from acamera position at an angle directly parallel to the ground.

In FIGS. 2A-B, an illustration of camera orientation parameters for adevice, such as mobile device 101 in FIG. 1, is shown. The cameraorientation parameters can be expressed in three dimensions, such as X,Y and Z axes, where the X axis represents a latitude position, the Yaxis represents a longitude position and the Z axis represents aposition of the device relative to a plane perpendicular to thedirection of gravity, e.g., ground. It shall be assumed for ease ofunderstanding and not limitation that the camera angle is fixed relativeto the orientation of the device 101. In that regard, FIG. 2Aillustrates a potential pitch angle of the device 101 (as seen lookingtowards the left side of the device) relative to the ground).

FIG. 2B illustrates a potential latitude/longitude angle of the device101 (as seen looking down towards the top side of the device), e.g., thecamera direction in which the camera points relative to the latitude andlongitude. Collectively, the pitch and latitude/longitude angle define acamera pose or location and orientation. The roll (rotation about the Yaxis of device 101), yaw/azimuth and/or altitude may also be captured.This and other image-related information may be outputted as numericalvalues by an accelerometer (not shown) or other component in the device101, used by the device's processor, and stored in the memory of thedevice 101.

Once the images are captured, they can be used to access and update a 3Ddata model associated with the location. For example, an application maybe employed to identify information about the images, such as a locationand visual orientation of the objects in the images. Then, relevantinformation concerning the images may be provided to a system foridentifying a corresponding data model of a location where the imageswere taken. An example of a system for processing these types of imagesused for updating a data model is further described below with respectto FIG. 3.

FIG. 3 illustrates an image processing system 300 for receiving a numberof input images 308, analyzing the imagery for relevant information andupdating a data model associated with a location depicted in the images308. As shown, the system 300 may have a number of filtering modules toidentify a location and a visual orientation associated with the inputimages 308. For example, the modules may include a location match filter310 to determine a location associated with an input image and a visualorientation filter 312 to determine an orientation of objects depictedin the image. Other filter modules (not shown) may also be employed. Thefilter modules may employ different techniques for analyzing images. Forinstance, the visual orientation filter 312 may include software foranalyzing an orientation of objects depicted in an input image.

The modules are shown as being connected to one or more data modeldatabases 314. These databases may store data models associated with alocation that may correspond to where the input images 308 werecaptured. These modules may operate in series or parallel fashion.Another module 316 of system 300 may use the results of the filteringmodules, for example, to update, correct and add to the data models inthe data model databases 314 that are associated with a location.

To locate a data model that corresponds to the input images, system 300may use the results of the location match filter 310. For example, thesystem 300 may select a data model that includes positioning informationlocated within an overlap area of the positioning information associatedwith input images 308. Thereafter, image projections associated with thedata models may be used to reconstruct a target image that depicts aparticular view of the location represented by the data model. Thisprocess for reconstructing a target image based on a data model isdescribed in further detail below with respect to FIG. 4.

FIG. 4 illustrates of aspects 400 of the image processing system 300 ofFIG. 3 as described above. As shown, the 3D geometry 416 associated witha data model may include a number of image projections 416A-E. Eachimage projections may include several related images that were captured,for example, in a video segment. When the image projections 416A-E areassembled using information stored with each related image, a targetimage 426 may be reconstructed depicting objects (e.g., building102-106) at the location where the related images were taken. In orderto determine whether a data model corresponds to a set of input imagescaptured at a location, the input images may be compared to the 3Dgeometry 416 associated with the data model.

FIG. 5 illustrates an image comparison technique 500. As shown, one ormore of the input images 508 may be compared to image projections of a3D geometry 416. A correspondence between the input images 508 and theimage projections may be determined based on information associated withboth images. This information can include visual orientation informationthat may indicate a visual orientation of objects depicted in theimages. If the visual orientation information associated with the inputimages 508 and image projections agree, then one or more of the inputimage 508 corresponds to one or more images from the image projections.

In some aspects, a further correspondence between the input images 508and image projections may be determined by analyzing a visualcorrelation between objects depicted in both images. In that regard,certain objects depicted in the input images 508, such as cars andpeople, may be removed or otherwise ignored or the input images 508 maybe preprocessed in order to simplify comparison of the input images 508with the image projections.

If there are no corresponding images determined between the input images508 and image projections of the 3D geometry 416, then the input images508 may be discarded or in some instances they can be used to model newimages of the location that were not previously integrated into the datamodel. If a correspondence between the input images 508 and imageprojections of the 3D geometry 416 is determined, then the correspondingimages are analyzed to locate a number of change regions.

FIG. 6 illustrates an image analysis technique 600. As shown, a numberof change regions 611-615 and 621-625 may be determined amongst theinput images 508 and image projections of the 3D geometry 416. Eachchange region represents a candidate area indicating an area of visualdifferent between corresponding areas of the input images 508 and theimage projections. For example, as shown in FIG. 6, change region 615from the input images 508 may correspond to an particular area of theimage projections of the 3D geometry 416 represented by change region625.

A score for the corresponding change regions may be determined based onthe image analysis. For example, the score may be based on a correlationbetween differences in visual features of objects depicted in thecorresponding images. The score can be further or alternatively based ona correlation between other image information and objects depicted inthe images or other conventional image analysis methods. If the scorefor a particular corresponding change region meets a certain level, itcan be determined that a particular area of the location covered by thechange region has significantly changed.

If a count of the change regions determined by the image analysis isbelow a pre-determined threshold level, then the data model may beupdated with one or more of the input images 508. This may ensure thatchanges to the 3D model are applied gradually. Alternatively, if a countof the change regions meets or exceeds the pre-determined thresholdlevel, the input images 508 will not be used to update the data modeland may be discarded.

FIG. 7 illustrates an updating technique 700 that may be used, forexample, to update a data model of a location. To update a data model ofa location stored in a data model database 314, one more input images508 may be integrated into a 3D geometry of images 416 associated withthe data model. As discussed above, this update may be based on thevisual orientation and positioning information associated with the inputimages 508 and similar information for corresponding images from the 3Dgeometry 416. In some instances, an alignment of the input images 508may be adjusted so that the images can be seamlessly integrated into thedata model. For instance, an alignment of the input images 508 mayinvolve updating and/or correcting a viewing angle of the input images508 to correspond with image projections of the 3D geometry 416associated with the data model.

While it can be possible to perform aspects of the image processing asdescribed above with an application installed locally on a mobiledevice, such as mobile phone, it may be more feasible to do theprocessing remotely as well as locally (or some combination thereof).For instance, a remote computing device like an application server mayperform some or all aspects of the image processing.

FIG. 8 illustrates an example of a system 800 including computingdevices for performing aspects of the present disclosure. As shown,system 800 depicts various computing devices that can be used alone orin a networked configuration. For example, this figure illustrates acomputer network having a plurality of computers 802 and 820 as well asother types of mobile devices like a mobile phone 830, a PDA 840 and alaptop/netbook 850. These various devices may be interconnected via alocal bus or direct connection 818 and/or may be coupled via acommunications network 895 such as a LAN, WAN, the Internet, etc. andwhich may be wired or wireless.

Each computing device may include, for example, user input devices suchas a keyboard 824 and mouse 826 and/or various other types of inputdevices such as pen-inputs, joysticks, buttons, touch screens, etc., aswell as a display 822, which could include, for instance, a CRT, LCD,plasma screen monitor, TV, projector, etc. Each computer 802 and 820 maybe a personal computer, application server, etc. By way of example only,computer 820 may be a personal computer while computer 802 may be aserver. Databases such as data model database 314 are accessible to oneor more of the servers or other devices of system 800.

FIG. 9 is a block diagram of a system 900, which may be used to maintaina data model database as described herein. As shown, the system 900includes a server 902 coupled to a network 995 and a number of clientdevices, such as client devices 930 and 940, capable of communicatingwith the server 902 over the network 995. The server 902 may include aprocessor 904, memory 906, and other components typically present ingeneral purpose computers.

The memory 906 of server 902 may store information that is accessible bythe processor 904, including instructions 908 that may be executed bythe processor 904, and data 912. The memory 906 may be of a type ofmemory operative to store information accessible by the processor 904,including a non-transitory computer-readable medium, or other mediumthat stores data that may be read with the aid of an electronic device,such as a hard-drive, memory card, read-only memory (“ROM”), randomaccess memory (“RAM”), digital versatile disc (“DVD”) or other opticaldisks, as well as other write-capable and read-only memories. Thesubject matter disclosed herein may include different combinations ofthe foregoing, whereby different portions of the instructions 908 anddata 912 are stored on different types of media.

Although FIG. 9 functionally illustrates the processor 904 and memory906 as being within the same block, the processor 904 and memory 906 mayactually include multiple processors and memories that may or may not bestored within the same physical housing. For example, some of theinstructions 908 and data 912 may be stored on removable CD-ROM andothers within a read-only computer chip. Some or all of the instructionsand data may be stored in a location physically remote from, yet stillaccessible by, the processor 904. Similarly, the processor 904 mayactually comprise a collection of processors, which may or may notoperate in parallel.

Data 912 may be retrieved, stored or modified by processor 904 inaccordance with the instructions 908. For instance, although the presentdisclosure is not limited by a particular data structure, the data 912may be stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents, orflat files. The data 912 may also be formatted in a computer-readableformat such as, but not limited to, binary values, ASCII or Unicode. Byfurther way of example only, the data 912 may be stored as bitmapscomprised of pixels that are stored in compressed or uncompressed, orvarious image formats (e.g., JPEG), vector-based formats (e.g., SVG) orcomputer instructions for drawing graphics. Moreover, the data 912 maycomprise information sufficient to identify the relevant information,such as numbers, descriptive text, proprietary codes, pointers,references to data stored in other memories (including other networklocations) or information that is used by a function to calculate therelevant data.

Data model databases 914 of server 902 may store data modelsrepresenting a 3D model of a location, which may be transmitted toclient devices 930 and 940. Each data model may be associated with a 3Dgeometry of images that include images captured at the location. Imageprojections from the 3D geometry may be used to reconstruct a targetimage that depicts object at that location. The data model databases 914can include information relevant the 3D geometry of images, such asvisual orientation information, latitude/longitude coordinatesrepresenting locations where the images were captured as well as othertypes of relevant data. Although the subject matter disclosed herein isnot limited to a particular positional reference system, for ease ofunderstanding latitude/longitude positions may be used when referencinglocations associated with data models.

The server 902 may be at one node of network 995 and capable of directlyand indirectly communicating with other nodes of the network 995. Forexample, the server 902 may include a web server that may be capable ofcommunicating with client devices 930 and 940 via network 995 such thatit uses the network 995 to transmit and display information to a user ona display 941 of the client device 940. Server 902 may also include aplurality of computers, e.g., a load balanced server farm, that exchangeinformation with different nodes of a network for the purpose ofreceiving, processing and transmitting data to client devices 930 and940. In this instance, the client devices 930 and 940 will typicallystill be at different nodes of the network 995 than the computerscomprising server 902.

The network 995, and intervening nodes, may include variousconfigurations and protocols including the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, WiFi (e.g., 802.11, 802.11b, g, n, or othersuch standards), HTTP, and various combinations of the foregoing. Suchcommunication may be facilitated by a device capable of transmittingdata to and from other computers, such as modems (e.g., dial-up, cableor fiber optic) and wireless interfaces.

Although certain advantages are obtained when information is transmittedor received as noted above, other aspects of the subject matterdisclosed herein are not limited to a particular manner of transmissionof information. For example, in some aspects, information may be sentvia a medium such as a disk, tape or CD ROM. Yet further, although somefunctions are indicated as taking place on a single server having asingle processor, various aspects may be implemented by a plurality ofservers, for example, communicating information to client devices 930and 940 over network 995.

Each client device 930 and 940 may be configured similarly to the server902, with a processor 942, memory 944, instructions 946, data 948 andall of the internal components normally found in a personal computer. Byway of example only, the client device 940 may include a centralprocessing unit (CPU), display device 941 (for example, a monitor havinga screen, a projector, a touch-screen, a small LCD screen, a television,or another device such as an electrical device that is operable todisplay information processed by the processor 612), CD-ROM, hard-drive,user input (not shown), such as a keyboard, mouse, touch-screen ormicrophone, speakers, modem and/or network interface device (telephone,cable or otherwise) and all of the components used for connecting theseelements to one another.

The client device 940 may be a computing device. For example, clientdevice 940 may be a laptop computer, a netbook, a desktop computer, anda portable personal computer such as a wireless-enabled PDA, a tablet PCor another type of computing device capable of obtaining information viaa network like the Internet. Although aspects of the disclosuregenerally relate to a single client device 940, the client device 940may be implemented as multiple devices with both portable andnon-portable components (e.g., software executing on a rack-mountedserver with an interface for gathering location information).

Although the client devices 940 may include a full-sized personalcomputer, the subject matter of the present disclosure may also be usedin connection with mobile devices capable of wirelessly exchanging data.For example, client device 940 may be a wireless-enabled mobile device,such as a Smartphone, or an Internet-capable cellular phone. In eitherregard, the user may input information using a small keyboard, a keypad,a touch screen or other means of user input. In various aspects, theclient devices and computers described herein may comprise a devicecapable of processing instructions and transmitting data to and fromhumans and other devices and computers.

The client device 940 may include a geographic positioning component947, such as circuits, to determine a geographic location of the device940. For example, the client device 940 may include a GPS receiver todetermine the device's latitude and longitude position. As the clientdevice 940 changes location, for example, by being physically moved, theGPS receiver may determine a new current location. By way of exampleonly, the component 947 may include software for determining theposition of the device based on other signals received at the clientdevice 940, such as signals received at a cell phone's antenna from oneor more cell phone towers if the mobile device is a cell phone. In thatregard, the provision of location identification data may occurautomatically based on information received from such a component.

As shown in FIG. 9, the client device 940 may also include an imagecapture module 945. The image capture module 945 can be used to capturevideo and/or still images of an object. The image capture module 945 maybe a software module operable in conjunction with a video camera or mayinclude a video graphing device, such as a video digital camera havingimage processing components. For example, the client device 940 may beconnected to a video digital camera that can operate in conjunction withthe image capture module 945. The image capture module 945 can alsooperate in conjunction with other image capturing systems known in thearts such as a digital camera with image and/or video capturecapabilities, a camera in a mobile phone, a video camera or otherdevices with image capturing features.

To determine the direction in which the client device 940 is oriented,for example, when capturing an image with the image capture module 945,the client device 940 may include one or more sensors 943 such as anaccelerometer, gyroscope, compass, or any combination of these. Forexample, an accelerometer may be used to detect an orientation of theclient device 940. By way of example only, the client device 940 may useinput from the accelerometer to determine the client device's pitch, yawor roll (or changes thereto) relative to the direction of gravity or aplane perpendicular thereto. In that regard, it will be understood thata client device's provision of orientation data as set forth herein maybe provided automatically to the client device 940. The orientation datareceived from the one or more sensors 943 may be used in various ways.For example, the orientation data may be associated with images capturedby the image capture module 945. This orientation data may indicate avisual orientation of captured images with respect to the direction ofthe client device 940 when the images were taken at a location.

The instructions 946 of the client device 940 may be a set ofinstructions to be executed directly (such as machine code) orindirectly (such as scripts) by the processor. In that regard, the terms“instructions,” “steps” and “programs” may be used interchangeablyherein. The instructions 616 may be stored in object code format fordirect processing by the processor, or in another computer languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

In order to facilitate the 3D data model updating operations of system900, the server 902 may further include a change region detector 909 fordetecting an area of visual difference between input images andcorresponding images from a data model and a 3D image integrator forintegrating the captured images into the data model. The functionally ofthese modules can exist in a fewer or greater number of modules thanwhat is shown, with such modules residing at one or more computingdevices, which may be geographically dispersed. The modules may beoperable in conjunction with client device 940 from which it may receiveimages depicting locations and relevant information regarding thoseimages. Thereupon, these images may be used to update and maintain datamodels associated with the locations.

Techniques for maintaining a data model of a location, for example usingsystem 900 discussed above, will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, as mentioned above, variousoperations can be handled in a different order or simultaneously, andoperations may be added or omitted.

FIG. 10 is a flow diagram 1000 depicting an example of some of theaspects described above. At stage 1010, a user may be prompted tocapture a video segment at a location. For example, an applicationinstalled on a user's mobile device may prompt the user to take a shortvideo segment at a location. This video may include images depictingobjects at the location that may be of interest to the user.

At stage 1020, the video segment captured at stage 1010 may be received.This video segment may include visual orientation and positioninginformation associated with the location. For example, the positioninginformation may indicate a latitude/longitude position associated with alocation the segment was captured. The visual orientation informationmay indicate a visual orientation of objects depicted in the images withrespect to the direction of a camera that may be associated with theuser's mobile device.

At stage 1030, a data model depicting the location may be accessed. Forexample, a data model may be selected from a database where thepositioning information associated with the data model overlaps with thepositioning information associated with the video segment.

At stage 1040, one or more change regions may be detected betweencorresponding images from the video segment and the data model. Eachchange region represents a candidate area indicating an area of visualdifferent between images of the video segment and image projections of a3D geometry associated with the data model.

At stage 1045, it is determined whether a count of the candidate changeregions is below a threshold value. If the count of the candidate changeregions is below a threshold value, the method 1000 may automaticallystop. Otherwise, method 1000 may continue onto stage 1050.

At stage 1050, the data model may be updated with at least part of thevideo segment based on the visual orientation and positioninginformation associated with the video segment. For example, one or moreof the images from the video segment may be integrated into a 3Dgeometry of images associated with the data model.

As these and other variations and combinations of the features discussedabove can be utilized without departing from the disclosure as definedby the claims, the foregoing description of the embodiments should betaken by way of illustration rather than by way of limitation of thedisclosure as defined by the claims. It will also be understood that theprovision of examples of the disclosure (as well as clauses phrased as“such as,” “e.g.”, “including” and the like) should not be interpretedas limiting the disclosure to the specific examples; rather, theexamples are intended to illustrate only some of many possibleembodiments.

1. A computer-implemented method, comprising: obtaining, using one ormore computing devices, a plurality of images depicting a location, theplurality of images associated with positioning data representing avisual orientation and position of each image with respect to thelocation; identifying, by the one or more computing devices, datarepresenting a 3D geometry of objects associated with the location;comparing, by the one or more computing devices, visual features ofobjects depicted in one or more corresponding areas between images fromthe plurality of images and image projections of the 3D geometry; andupdating, by the one or more computing devices, the data representingthe 3D geometry based at least in part on the comparison between visualfeatures in the images from the plurality of images.
 2. The method ofclaim 1, wherein further comprising receiving an indication that a userarrived at the location.
 3. The method of claim 1, wherein identifyingthe data representing the 3D geometry further comprises determiningwhether positioning data associated with the plurality of imagesoverlaps with positioning data for the 3D geometry.
 4. The method ofclaim 1, further comprising determining a visual correlation between oneor more objects depicted in the images from the plurality of images andone or more objects depicted in the image projections.
 5. The method ofclaim 1, wherein the comparing further comprises detecting visualdifferences between features of objects depicted in a given area of aparticular image and features of objects depicted in a correspondingarea of the image projections.
 6. The method of claim 1, furthercomprising determining an alignment of an integrated image from theplurality of images based on a corresponding image projection from the3D geometry.
 7. The method of claim 6, further comprising adjusting thealignment of the integrated images in the 3D geometry using the visualorientation information.
 8. A non-transitory computer readable medium,storing instructions that, when executed by one or more processors,cause the one or more processors to perform a method, the methodcomprising: obtaining, using one or more computing devices, a pluralityof images depicting a location, the plurality of images associated withpositioning data representing a visual orientation and position of eachimage with respect to the location; identifying, by the one or morecomputing devices, data representing a 3D geometry of objects associatedwith the location; comparing, by the one or more computing devices,visual features of objects depicted in one or more corresponding areasbetween images from the plurality of images and image projections of the3D geometry; and updating, by the one or more computing devices, thedata representing the 3D geometry based at least in part on thecomparison between visual features in the images from the plurality ofimages.
 9. The computer readable medium of claim 8, wherein the methodfurther comprises receiving an indication that a user arrived at thelocation.
 10. The computer readable medium 8, wherein identifying thedata representing the 3D geometry further comprises determining whetherpositioning data associated with the plurality of images overlaps withpositioning data for the 3D geometry.
 11. The computer readable medium8, wherein the comparing further comprises detecting visual differencesbetween features of objects depicted in a given area of a particularimage from the plurality of images and features objects depicted in acorresponding area of the image projections.
 12. The computer readablemedium 11, further comprising determining an alignment of an integratedimage from the plurality of images based on a corresponding imageprojection from the 3D geometry.
 13. The computer readable medium 12,further comprising adjusting the alignment of the integrated images inthe 3D geometry using the visual orientation information associated thecorresponding image projection from the 3D geometry.
 14. A system,comprising: a memory; and one or more processors coupled to the memory,the one or more processors being configured to: obtain a plurality ofimages depicting a location, the plurality of images being associatedwith positioning data representing a visual orientation and position ofeach image with respect to the location; identify data representing a 3Dgeometry of objects associated with the location; compare visualfeatures of objects depicted in one or more corresponding areas betweenimages from the plurality of images and image projections of the 3Dgeometry; and update the data representing the 3D geometry based atleast in part on the comparison between visual features in the imagesfrom the plurality of images.
 15. The system of claim 14, wherein toobtain the plurality of images, the one or more processors are furtherconfigured to receive an indication that a user arrived at the location.16. The system of claim 14, wherein to identify the data representingthe 3D geometry, the one or more processors are further configured todetermine whether positioning data associated with the plurality ofimages overlaps with positioning data for the 3D geometry.
 17. Thesystem of claim 14, wherein the one or more processors are furtherconfigured to determine a visual correlation between one or more objectsdepicted in the images from the plurality of images and one or moreobjects depicted in the image projections.
 18. The system of claim 14,wherein to compare visual features of objects, the one or moreprocessors are further configured to detect visual differences betweenfeatures of objects depicted in a given area of a particular image fromthe plurality of images and features of objects depicted in acorresponding area of the image projections.
 19. The system of claim 14,wherein the one or more processors are further configured to determinean alignment of an integrated image from the plurality of images basedon a corresponding image projection from the 3D geometry.
 20. The systemof claim 19, wherein the one or more processors are further configuredto adjust the alignment of the integrated images in the 3D geometryusing the visual orientation information associated with the pluralityof images.